Pictorial images are often classified by the particular event, subject or the like for convenience of retrieving, reviewing and albuming of the images. Typically, this has been achieved by manually segmenting the images, or by an automated method that groups the images by color, shape or texture in order to partition the images into groups of similar visual content. It is clear that an accurate determination of content would make the job easier. Although not directed to event classification, there is a body of prior art addressing content-based image retrieval and the content description of images. Some typical references are described below.
In U.S. Pat. No. 6,072,904, “Fast image retrieval using multi-scale edge representation of images”, a technique for image retrieval uses multi-scale edge characteristics. The target image and each image in the data base are characterized by a vector of edge characteristics within each image. Retrieval is effected by a comparison of the characteristic vectors, rather than a comparison of the images themselves. In U.S. Pat. No. 5,911,139, “Visual image database search engine which allows for different schema”, a visual information retrieval engine is described for content-based search and retrieval of visual objects. It uses a set of universal primitives to operate on the visual objects, and carries out a heterogeneous comparison to generate a similarity score. U.S. Pat. No. 5,852,823, “Image classification and retrieval system using a query-by-example paradigm”, teaches a paradigm for image classification and retrieval by query-by-example. The method generates a semantically based, linguistically searchable, numeric descriptor of a pre-defined group of input images and which is particularly useful in a system for automatically classifying individual images.
The task addressed by the foregoing three patents is one of image retrieval, that is, finding similar images from a database, which is different from the task of event clustering for consumer images, such as photo album organization for consumer images. The descriptors described in these patents do not suggest using foreground and background segmentation for event clustering. Most importantly, the segmentation of images into foreground and background is not taken into account as an image similarity measure.
Commonly-assigned U.S. Pat. No. 6,011,595, “Method for segmenting a digital image into a foreground region and a key color region”, which issued Jan. 4, 2000 to T. Henderson, K. Spaulding and D. Couwenhoven, teaches image segmentation of a foreground region and a key color backdrop region. The method is used in a “special effects” process for combining a foreground image and a background image. However, the foreground/background separation is not used for image similarity comparison.
Commonly assigned U.S. patent application Ser. No. 09/163,618, “A method for automatically classifying images into events”, filed Sep. 30, 1998 in the names of A. Loui and E. Pavie, and commonly-assigned U.S. patent application Ser. No. 09/197,363, “A method for automatically comparing content of images for classification into events”, filed Nov. 20, 1998 in the names of A. Loui and E. Pavie, represent a continuous effort to build a better system of event clustering for consumer images, albeit with different technical approaches. Ser. No. 09/163,618 discloses event clustering using date and time information. Ser. No. 09/197,363 discloses a block-based histogram correlation method for image event clustering, which can be used when date and time information is unavailable. It teaches the use of a main subject area (implemented by fixed rectangle segmentation) for comparison, but does not propose any automatic method of performing foreground/background segmentation, which would be more accurate than a fixed rectangle.
Two articles—one by A. Loui, and A. Savakis, “Automatic image event segmentation and quality screening for albuming applications,” Proceedings IEEE ICME 2000, New York, August, 2000 and the other by John Platt, “AutoAlbum: Clustering digital photographs using probabilistic model merging”, Proceedings IEEE Workshop on Content-based Access of Image and Video Libraries, 2000—specifically relate to event clustering of consumer images; however they do not look into regions of images and take advantage of the foreground and background separation. Loui and Savakis teach an event clustering scheme based on date and time information and general image content. Platt teaches a clustering scheme based on probabilistic merging of images. Both of them fail to address the foreground and background separation.
What is needed is a system for segmenting images into coarse regions such as foreground and background and deriving global similarity measures from the similarity between the foreground/background regions. Furthermore, such a system should not become confused by unnecessary details and irrelevant clusters in consumer images.